An Evaluation of Synthetic Speech Using the PESQ Measure

The paper presents experiments on the use of the perceptual objective measure – ITU-T Rec. P.862 Perceptual Evaluation of Speech Quality (PESQ), for the automatic evaluation of synthetic speech. The approach is based on the evaluation of the statistically significant correlation between the outputs of subjective and objective tests. We propose the following technique to evaluate the usage of the PESQ method for synthetic speech: Firstly, a list of the test words has to be defined for the entire language. Secondly, the tested synthesizers are used to generate synthetic speech signal for all the words in the list. Synthesizer engines of different quality were used for the generation of stimuli: LP synthesizer, RELP synthesizer and PSOLA synthesizer, both in female and male versions. We evaluated created stimuli by listening tests. Thirdly, the PESQ method with original human (reference) and synthesized (measured) recordings as inputs is used to evaluate the overall quality of the synthesized signals. Finally, a correlation of the resulting MOS and objective MOS scores is calculated for each voice. Our results indicate a strong correlation between the mentioned subjective and objective evaluation of the quality of synthetic speech. We plan to use the PESQ measure in automatic evaluation of new versions of synthetic voices, without a need of subjective tests. This approach can foster the life cycle of the development of new versions of synthetic voices tremendously. Using PESQ with “original voice” as reference represents a rapid and repeatable synthetic voice quality measurement technique that provides the developer with results in a few moments.